Bias in Political Communication Experiments

Research on political communication effects has enjoyed great progress over the past 20 years. A key ingredient underlying these advances is the increased usage of experiments that demonstrate how communications influence opinions and behaviors. A potential problem with nearly all of these studies, however, concerns the lack of attention to events that occur prior to the experiment—that is, “pretreatment events.” In this paper, the authors explore how and when the pretreatment environment affects experimental outcomes. They argue that ignoring pretreatment has led extant work to overstate the malleability of the mass public, miss the identification of potentially two groups of voters—what they call malleability reactive and dogmatic—and contradict macrolevel work on aggregate public opinion trends. We thank seminar participants at Harvard University and the University of Kansas for helpful comments. Over the last twenty years, scholars have made remarkable progress in understanding how mass communications shape the public’s opinions. The field has moved from being “one of the most notable embarrassments of modern social science” (Bartels 1993: 267) to introducing “compelling” concepts that have “had a major impact in political science and communications scholarship” (Iyengar 2010: 190). Indeed, researchers no longer ask whether communications shape opinions, but rather when and how. A bulk of the research on mass communication effects comes from experiments. A typical study randomly exposes some respondents to one message (e.g., a description of a hate group rally request framed as a free speech issue), and other respondents to a different message (e.g., a rally description using a public safety frame). When opinions of the groups differ, it is taken as evidence that communications affect opinions (see Nelson et al. n.d.). But just how much do these experiments – which have been conducted with a wide range of people on innumerable topics – reveal about the influence of political communication? One notable problem concerns timing and, specifically, what occurred before the experimental treatments (i.e., “pretreatment”). If the experiment explores a communication that regularly occurs in “reality” then reactions in the experiment might be contaminated by those “regular” occurrences prior to the experiment. For example, it could be that the aforementioned free speech frame registers no effect because it already moved the respondents before the experiment and one more exposure in the experiment does little. If, on the other hand, the experiment focuses on a communication that had not been previously experienced in “reality,” then one could question the relevance of the experiment. Gaines et al. (2007: 12) state that “Put simply, either there is a likelihood of contamination from real-world experience of the... [or the] experiment explores a nonexistent or politically irrelevant phenomenon.” Despite the potentially grave consequences of pretreatment effects – as raising serious questions about experimentally based inferences – there has been virtually no work on the topic (for a partial exception, see Slothuus 2009). In this paper, we provide what we believe is the first conclusive evidence 1 Experiments enable researchers to know with near certainty the communications to which respondents were exposed and that respondents did not themselves select those communications. Nelson et al. (n.d.: 3) state that experimentation and communication effects “seem made for each other.”